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Digital Image Processing Including Block Edges Filtering 



Background of tlie invention 



The present invention relates to digital image processing 
using block edges filtering. 

A large number of coding/decoding methods for full-motion 
digital images have been proposed in recent years. ' 

Because the available data rate in present transmission chan- 
nels and storage media is limited it is an object of these 
coding methods to reduce the data rate of the coded and 
transmitted or stored video signal. 

For the purpose of storing digital motion images on optical 
storage media, e.g. CD-ROM, MOD (Magneto Optical Disc), a 
maximum video data rate of about 1.2 Mbits/s is allowed. 

One general example of a system for coding and decoding digi- 
tal video signals is disclosed in "Scene Adaptive Coder", 

W.H. Chen, W.K. Pratt, IEEE Transactions on Communications, 
Vol . COM-32, No. 3, issued in March, 1984 in which scene 
adaptive coding/decoding of video signals is described, us- 
ing DCT (discrete cosine transform), a variable threshold 
for the DCT coefficients, quantization, Hxiffraan-coding emd a ' 
l^uffer in the coder and a buffer, Huffmann decoder, thresh— , 
old adding and inverse DCT in the decoder. 

In US Patent No. 4,785,349 in a simular way regions of a 
frame are coded separately. In addition, motion compensation 
is performed using this regions. 

In both systems the fullness of the buffer controls the 
threshold and the q[uantizer step size, respectively. It is 
to be made sure that the buffer will have no overflow or un— 
derflow within the present frame. This causes a varying 
quantizer step size and a consequently varying image quality 
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within each frame. But if blocks and regions, respectively, 
are coded euxd decoded including data rate reduction, arte- 
facts (blocking effects) will arise after decoding at the 
boundaries of these blocks and regions, respectively. 



Suinmary of the invention 



It is an object of the present invention to make artefacts 
at block edges invisibie in case of motion compensated digi- 
tal images and to improve the image quality of the decoded 
images. 

A sequence of digital images is coded using differential cod' 
ing. The nth image of the sequence is partitioned in blocks 
B(n,i,j) of pixels, where i and j are the frame coordinates 
of the left top pixel of the block and n is a frame number. 

Xhe temporal correlation between the image content of frame 
n and frame n-1 is used to code each block in frame n as a 
displacement of the most similar block in frame n-rl. 

V(n,x,y) is called the displacement of the block B(n,i,j) 
from frame n-1 to frame n. This displacement is used as a 
predictor for the motion compensation. If B'(n,i,j) is the 
decoded block B(n,i,j), then B'(n,i,j) = B’ (n-1, i+x, j+y) . 

V(n,x,y) is computed by means of a search in the decoded 
frame n-1 for a block, which minimizes the difference be- 
tween B(n,i,j) and B' (n-1, i+x, j+y ) . The steps for a coding 
loop are: 

(1) code frame n using motion compensation and the current 
predictor 

(2) decode frame n 

(3) filter decoded frame n 
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(4) select the decoded and filtered frame n and calculate 
the current predictor 

(5) go to step (1) to code fraune n+1 

The decoding loop is build accordingly in a symmetrical man- 
ner . 



The basic idea of the invention is to apply a filter within 
the decoding loop on the edges of the blocks to reduce the 
block effect. This filter improves the quality of the cur- 
i***^9® and reduces the propagation of blocking effect 
artefacts in subsequent images. 

Another aspect of the invention is to apply this filter to a 
block edge only if the two adjacent blocks have different 
displacements. By doing this the resolution of the image is 
enhanced. 



The advantages of the present invention will become more ap- 
parent from the following more detailed description when tak~ 
in conjunction with the accompanying drawings. 



®rief description of the drawings 



Fig. 1 
Fig. 2 
Fig. 3 
Fig. 4 



is a schematic block diagram of a coder 
is a schematic block diagram of a decoder 
shows an image with adjacent blocks of pixels 
shows a quantization matrix for luminance blocks 



Fig. 5 



shows a scanning order 




wo 91/14340 



PCT/EP91/00451 



- 4 - 



Fig. 6 shows a quantizer function graph 



Description of the preferred embodiments 



The images have a medium resolution, e.g. 704 * 288 pixels. 
The luminance component Y is coded for each of the pixels, 
v/hereas the color components U and V are coded in the form 
of macro-pixels, a macro— pixel being made of 4 luminance 
pixels. 

Each frame is divided into marcro-blocks of 16*16 luminance 
pixels. Each macro-block is composed of: 

- 4 blocks of 8*8 pixels for Y, 

- 1 block of 8*8 macro-pixels for U, 

- 1 block of 8*8 macro-pixels for V. 

A single 704 * 288-image not encoded consists of: 

(704 / 16) * (288 / 16) = 792 macro blocks, 

792 * 6 = 4752 blocks, 

- 4752 * 64 = 304128 bytes. 

Such a frame would occupy a size of about 300 KByte in the 
uncoded format, whereas the available bandwidth allows about 
(1.2 Mbit/s) / (8 * 25 /s) = 6 KByte per frame, i.e. less 
than 1/2 bit per pixel. A compression rate of eibout 1/50 is 
to be achieved. 

In Fig. 1 an digital image source 11 is connected to a frame 

memory (a RAM) 12 which stores frame n. Sec[uentially each 
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block of pixels is then send to a motion compensation cir- 
cuit 17. ■ vig 

As most of the time fraune n-1 is very close to frame n, only 
the differences between frames n-1 and n are encoded. This 
technique is called interframe coding (delta-coding) and re- 
sults in a relatively low -data rate. 

However in some cases it is necessary to encode a freune by . • ^ 
itself, e.g. when there is a "cut" in the image sequence. 

This technique is called intraframe coding. It is also neces-^' 
sary to use intraframe coding to allow non- sequential decod- 
ing, e.g. start decoding from various points within an image 

= • r - y ■; . 

sequence stored on a CD. But intraframe coding results in a 
higher data rate. 

Because in case of motion in the image sequence interframe ;r 
coding may yield no good c[uality of the coded image, moving 
pieces in the image sequence should be detected. Thereby the . 
differences between the moving pieces from fr 2 une n-1 and th<e 
according pieces from frame n are minimized and the inter- 
frame coding leads to improved image quality. The purpose of 
motion compensation is to build an image as close as possi- 
ble to frame n by only taking moving pieces out of frame n-1.' 

This is performed within motion compensation circuit 17 on a_ V*: 
block basis; the block B(n,i,j) of frame n located at frar>a . 
coordinates (i,j) is evaluated by copying the block 
B' (n-l,i+x, 3 +y) stored at frame coordinates (i+x, j+y) in a 
frame memory (a RAM) 142 which contains frame n-1. The dis- 
placement V(n,x,y) represents the apparent motion of the im- 
age at this point of screen. When using 8*8 pixel blocks for-.:.; 
motion compensation the displacement may have a precision of 
±1 pixel and a range of ±16 pixel. 
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In case of non-random block scanning the motion compensation 
vectors are delta coded. The delta vectors (dx, dy) range 
from -32 to -*-32 and the most frequent value encountered is 0 
(vector S6une as previous vector) . The delta coordinates are 
coded independently with adapted Huffman codes in Huff man 
coder 154. As the statistics for dx and dy are not identical 
two Huffman codes are used. 

Then the vectors are packed 4 by 4: As there is a big chance 
for dx and dy to be zero, the distribution of zero values 
among the 4 dx values is coded with an adapted Huffman code. 
Only the non-zero values of dx are coded after that (with 
another adapted Huffman code). The same technique is used 
for dy. 

In order to optimise the size of the code for the motion com- 
pensation vectors, a certain tolerance is accepted. If V is 
the target block (not coded), V is the, block obtained by us- 
ing the best motion compensation vector computed for V, and 
V'^ is the block obtained by using the previous motion com- 
pensation vector (the one which was used for the block left 
of V), the previous motion compensation vector will be used 
again, if E(V'^,V) < E(V',V) + E^ . 

By choosing an appropriate threshold E^, the number of delta 
vectors equal to zero can become very big, and the coding 
will be much more efficient because zero delta vectors are 
coded with very few bits. This leaves more room for the er- 
ror correction. 

Once a good approximation of frame n has been obtained from 
frame n-1 by using motion compensation, the residual pixel 
differences remain to be encoded. To have an estimation of 
the difference between two blocks and to decide for each 
block, if motion compensated interframe coding or intraframe 
coding yields better image quality in connection with a giv- 
en data rate, an error function (e.g. the quadratic error) 
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is evaluated in motion compensation circuit 17. Th.e equations - 
of the quadratic error is: 

7 7 

E(B,B') = 21 21 (P(iiOJ) - P' (ii, jj})* 

ii=0 jj=0 

where ii and Jj are the coordinates within the blocks 
B(n,i,j) and B' (n-1 , i+x, J +y) , respectively, of pixels P and v; 
P', respectively, and E(B,B' ) is an estimation of the differ- 
ence between block B(n,i,j) of frame n and block 
B' (n-1, i+x, j+y) of frame n-1. B(n,i,j) is the target block 
(not coded), B' (n-1, i+x, j+y) is the block obtained after 
interframe coding*. 

Motion compensation is worth being encoded only if: 

E(B,B*) < (1 / 2) * E(B,Bq) 

Bq is the block B(n-1, i+x, j+y) obtained only with motion com- 
pensation (not interframe coded). 

If motion compensation is calculated for 8*8 pixel blocks 
'^i'thin a range of +16 pixels, the research for the best 
matching block is performed in a 40*40 pixels area (33*33 
possible displacements). A brute force search is performed . 
among all the possible displacements (x,y) and the displaceT„. 
raent which corresponds to the minimum mean absolute error is 
chosen. This vector is transferred to a Huffman coder 154. 
Motion compensation circuit 17 may contain a memory in which 
this 40*40 pixel area is stored. 



The equation of the mean absolute error Era is: 
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7 7 




Em(B,B') = 


^ z: 

ii=0 jj=0 


|P(ii, jj) - P' (ii, j j) 1 , 



where ii and j j are the coordinates within the blocks 
B(n,i,j) and B’ (n-l,i+x, j+y) , respectively, of pixels P and 
p' , respectively. B(n,i,j) is the target block (not coded), 

B' (n-l,i+x, j+y) is the block obtained after interframe cod- 
ing. The Y component pixel values are used. 

In theory the motion compensation should be performed in the 
decoded frame n-1 for that is the one used at decoding time 
to built the next frame, but practically the source frame 
n-1 is used instead of the decoded one. 

Displacements obtained by using source images are easier to 
encode efficiently, because of the lower level of noise and 
of the better quality of the source images compared with de- 
coded images. Random noise tends to generate a random factor 
in the displacements. 

Motion compensation in principle is also described in US Pat- 
ent No. 4,785,349, colimns 27 - 29. 

In case of intraframe coding the block B(n,i,j) of pixels of 
frame n is coded directly, in case of interframe coding the 
block B' (n-1, i+x, j+y) is subtracted from block B(n,i,j) in 
motion compensation circuit 17 and then coded. Frame memory 
142 receives addresses from motion compensation circuit 17- 

The resulting block of pixels or pixel differences, respec- 
tively, is transformed in a DCT circuit 151. 

Each 8*8 block of values representing spatial data (such as . 
the Iximinance component Y) is transformed by a DCT into an 
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8*8 block of spectral coefficients representing the frequen- 
cy distribution in both, horizontal and vertical direction. 
The range of the coefficients in the transformed block is 
eight tiroes the range of the values in the source block. The 
top-left coefficient of the transformed block is particular; 
it is the average value of the source block (multiplied by 
eight) . This coefficient is called DC value, the other coef- 
ficients are called AC values. It appears that the DC coeffi- 
cient is very important, and that even if a high quanti- 
zation level is requested, there should be a minimum error 
on this Value since an error on the average produces very . 
noticeable artefacts (blocking effects) . 

Before being encoded further, the block is quantized in 
quantizer 152 by an algorithm chosen to trade off perceived 
image quality against bit rate. The algorithm of quantizer, , 
152 is described by the following equations: 



X - T 

y = + 1 for X _> T, 

9 

y = 0 for -T < x < T, 

X + T 

y = - 1 for X <, -T, 



9 

where x is a DCT coefficient of the transformed block, y is 
the qpiantized coefficient, T is the threshold at which the 
output is forced to zero and g is the quantization step 
size. The values of T and g are chosen according to the 
availedale transmission or storage bandwidth. The according 
quantizer function graph is shown in Fig. 6. 
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The quantizer step size g depends on the position of the co- 
efficient in the transformed block: high frecfuency coeffi- 
cients will be gueuitized coarser than low frequency coeffi- 
cients because lower frequency DCT coefficients are relative- 
ly less important in the perception of the sxibjective quali- 
ty of the decoded image. The quantizer step size is obtained 
for each position in the block by scaling with a predefined 
quantization matrix Q. As an example, the quantization ma- 
trix used for Y is shown in figure 4. A different 
quantization matrix is used for U and V. 

A minimum value should be defined for g, to guarantee that 
the quantized frequency coefficients are always in the range 
covered by the code used to encode them. As for threshold T, 
one may use T = g / 2 . 

In order to simulate the behaviour and the decoding error of 
a decoder (Fig. 2) and to correct the coding calculation 
within the coder (Fig- 1) accordingly, a feedback loop is 
built consisting of a de-quantizer 156, a DCT circuit 157 
and a filter circuit 158. De-quantizer 156 performs the ac- 
cording inverse function of quantizer 152 and DCT circuit 
157 an inverse DCT. The filter circuit 158 is used in order 
to eliminate source and cpaantization noise, glitches, isolat- 
ed pixels, to reduce high frequencies which are hard to code 
and to reduce the propagation of block effect artefacts in 
subsequent images. The filter may be a median filter or a 
simple convolution filter, both two-dimensional with a 3*3 
pixels kernel, this kind of filters are well known by a man 
skilled in the art. 

The filtered blocks B'(n-l,i,j) at the output of filter cir- 
cuit 158 are stored in frame memory 142 which is also con- 
nected to motion compensation circuit 17. By this way the 
feedback loop is closed. 
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The quantized block consists usually of few important spec- 
tral components (statistically concentrated in the low hori- ■“ 
zontal and vertical frequencies), separated by a lot of ze- 
ros. Therefore the quantized DCT coefficients in the block 
folded, thereby transformed in a linear list con-, 
taininq alternatively a run of r zeros ( r may be equal to O) 
and a non zero coefficient and are then encoded with vari— 
able wordlength (runlength coding) together with the dis- 
placements in the Huffman coder 154. 

The folding of each of the matrices of 64 DCT coefficients t 
is performed in a sorter 153 accordingly to the order shown 
in Fig. 5. The sorting starts with coefficient No. 1, which 
is the DC coefficient, is continued in a viceversa diagonal ^ 
manner and ends with coefficient No. 64. This kind of sort- 
ing is called 'zigzag scanning' . After the last coefficient 
of the present block an End-of-Block code (EOB) is added. 

The zigzag order can be chosen in order to increase the prob- 
of having a very long run of zero coefficients \mtil 
the end of the block. Because the EOB code is a frequent 
e\rent^ it: is coded with. £ew bits. 

The optimal zigzag order depends on the quantization method 
and of the code used subse< 5 uently . 

Zigzag scanning and runlength coding are described in princi- .■ 
pie in the above mentioned article "Scene Adaptive Coder" by 
Chen and Pratt. ‘ 

code for encoding the runs of zeros with a meuci — 
mum efficiency contains 64 possible runs of r zeros (r rang- 
ing from O to 63). An additional code is reserved for the 
EOB sign. The. code defined for each run of zeros is totally 
independent of the value that follows the run. 
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Another Huffman code encodes the non-zero coefficients with 
a maximum efficiency. The range covered by this code is 
1-256,-11 and (+l,+256l. 

As quantized non-zero coefficients have an important proba- 
bility to be ±1, particularly in the higher frequency coeffi- 
cients because they are more strongly quantized, a special 
code has been reseirved to code a terminal run of ±1 values. 
The signs of the non-zero coefficients are always coded with 
one bit, as their event may be considered as random with a 
50% probability. 

The Huffman coded data are stored in the buffer within buff- 
er circuit 155 with a variable data rate and read out with a 
constant data rate. A channel coder 16 receives these data 
together with other signals. 

In order to guaranty the data rate and to avoid buffer over- 
flow or underflow the quantization step size of quantizer 
152 and de-quantizer 156 is adapted by means of a quanti- 
zation level, which is recalculated after each block has 
been encoded within buffer circuit 155 as a function of the 
amount of the already encoded data for the frame, compared 
with the total buffer size. This way quantization level can 
advantageously be recalculated by the decoder and does not 
need to be transmitted. 

The scauuiing of the blocks, e.g. 32, 33, 34, 35, 36, within 
the present frame 31 is performed normally row by row and 
column by column. 

Advantageously the scanning may be performed in a pseudo-ran- 
dom fashion to have a better estimation of the overall 
quantization level for a frame to achieve a certain data 
rate. The pseudo— random block scanning order can be changed 
for each frame to get even better results. In this case the 
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kind of block scanning will be transmitted and stored, re- 
spectively, too. 

The channel coder 16 adds the inter/intra— information and 
the Huffman coded motion compensation information and e.g. . 
pseudo-random address information, audio signals, error cor- , 
rection information and sync words to the video data stream. 



Clock Generator 191 supplies control circuit 192 with clock 
pulses. Control circuit 192 supplies all the other circuits ’ 
in the coder with appropriate control signals. 

In Fig. 2 a channel decoder 26 receives the storage media or ’ 
transmission channel data stream. This data stream is split 
into luminance and chrominance signals (Y, U and V), motion 
compensation information, inter/intra-decision information, 
audio signals, sync words and, if required, pseudo random 
block addresses. In addition error correction is performed *• 
within channel decoder 26. 

The video signal, e.g. the lumin 2 uice component, is fed with, 
a constant data rate together with the Huffman coded dis- 
placements into buffer circuit 255 and transferred with a 
varicJole data rate from this buffer to a Huffman decoder 
254. In this circuit the according decoding of the different . - 
Huffman codes, the special code for a terminal run of ±1 val- 
ues and EOB code is performed. 

In the inverse sorter 253 an inverse scanning according to 
sorter 153 and Fig. 5 is performed. At the output are blocks 
of gu5m.tized 8*8 DCT coefficients available. In de-quantizer 
252 the coefficients are expanded approximately to their val- 
ue at the input of quantizer 152 in the coder. This includes . 
an Inverse scaling according to quantization matrix Q (Fig. . ' 
4) . The quantization level in de-quantizer 252 is controlled 
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Claims 



1. Method of encoding digital image sequences, comprising 
the steps of : 

—scanning of frames of a digital video signal of a type 
in which each frame thereof comprises a plurality of 
blocks of pixels to get these blocks of pixels, 
-estimating motion vectors and calculating block errors 
(quadratic error and/or mean eibsolute error) , 

-providing intrafraucie-blocks of pixel values or motion 
compensated interframe-blocks of pixel difference val- 
ues depending on the result of the block error calcula- 
tion and providing an intra/inter decision signal, 
—transforming the blocks of pixel values and pixel dif- 
ference values, respectively, with a DCT to blocks of 
coefficients, 

-coding the blocks of coefficients and the motion vec- 
tors in a Huffman coder with variable word- length, 
thereby providing groups of coded coefficients, 
—quantizing each group of coded coefficients individual- 
ly by the use of a quantization level and means for 
quantizing, each coded coefficient of the group individu- 
ally by the use of a quantization matrix, controlled by 
the fullness of a buffer. 

2. Method as defined in claim 1, comprising the step of: 
-coding the motion vectors in the form of difference 
vectors with respect to the according foregoing vector. 

3. Method as defined in claim 1 or 2, comprising the steps 
of: 

-inserting. a feedback loop, consisting of de— quantizing 
and inverse DCT transforming after the quantizing step. 
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-using -the resulting decoded coefficient blocks for mo- 
tion estimation, error calculation and motion compensa- 
tion. 

4. Method of decoding digital image sequences which are 
encoded as defined in claim 1, 2 or 3, comprising the 
steps of : . 

-dec[uantizing the groups of coded coefficients which 
belong to blocks of pixels individually by the use of 
the encoding quantization level and de-quantizing each 
coded coefficient of each group individually by the in- 
verse use of the encoding quantization matrix, con- 
trolled by the fullness of a buffer, “ 

-decoding the groups of coded coefficients and motion 
vectors in a Huffman decoder with variable wordlength, 
thereby providing blocks of decoded coefficients, 
-transforming the blocks of decoded coefficients with 
an inverse DCT to pixel values and pixel difference val- 
ues, respectively, 

-generating blocks of pixels out of intra-frame-blocks , 
of pixel values and motion compensated interframe- 
blocks of pixel difference values, respectively, depend- 
ing on eua intra/inter decision signal, thereby evaluat- 
ing the motion vectors, 

-filtering the pixels at the boundaries of the blocks 
of pixels, 

-inserting the blocks of pixels in frames of a digital 
video signal. 

5. Method as defined in claim 4, comprising the step of: 
-filtering the pixels at the according boundaries of 
the blocks of pixels only if the motion vectors for the 
according blocks are not nearly identical. 
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6- Method as defined in one or more of the claims 2 to 5, 
comprising the step of: 

-filtering the blocks of the decoded coefficients in 
two spatial directions. 

7. Apparatus for encoding digital image sequencesy compris- 
ing; 

-means for providing a compressed digital video signal 
of a type in which each frame thereof comprises a plu- 
rality of blocks of pixels, 

-means for scanning the frames to get the blocks of 
pixels, 

-means for estimating motion vectors and means for cal- 
culating block errors, 

-means for coding the motion vectors in the form of dif- 
ference vectors with respect to the according foregoing 
vector, 

-means for providing intraframe-blocks of pixel values 
or motion compensated interframe-blocks of pixel differ- 
ence values depending on the result of the block error 
calculation auid means for providing an intra/inter deci- 
sion signal, 

-me 2 uas for de-quantizing and inverse DCT transforming 
the quantized blocks of coefficients in a feedback loop 
where theses blocks form an input signal for the means 
which perform motion estimation, error calculation and 
motion compensation, 

-means for transforming the blocks of pixel values and 
pixel difference values, respectively, with a DCT to 
blocks of coefficients, 

-means for coding the blocks of coefficients and the 
motion vectors in a Huffman coder with variable wprd- 
length, thereby providing groups of coded coefficients, 
-means for quantizing each group of coded coefficients 
individually by the use of a quantization level and 
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means for quantizing each coded coefficient of the 
group individually by the use of a quantization matrix, 
controlled by the fullness of a buffer. 

8. Apparatus for decoding digital image sequences, compris- 
ing: 

-means for de-quantizing groups of coded coefficients 
which belong to blocks of pixels individually by the 
use of an encoding quantization level and means for de— 
quantizing each coded coefficient of each group individ- 
ually by the inverse use of an encoding quantization 
matrix, controlled by the fullness of a buffer, 

-means for decoding the groups of coded coefficients 
and motion vectors in a Huffman decoder with variable 
wordlength, thereby providing blocks of decoded coeffi- 
cients, 

-means for transforming the blocks of decoded coeffi- 
cients with an Inverse DCT to pixel values and pixel 
difference values, respectively, 

-means for generating blocks of pixels out of intra- ^ • 

freune-blocks of pixel values and motion compensated 
interframe-blocks of pixel difference values, respec- 
tively, depending on an intra/inter decision signal, 
thereby evaluating the motion vectors, 

-filtering mesuis for the pixels at the according botinda- 
ries of the blocks of pixels which filter the according 
pixels if the motion vectors for the according blocks 
are not nearly identical, 

-means for inserting the blocks of pixels in frames of 
a digital video signal in a pseudo-random order. 

. Apparatus as defined in claims 7 or 8, comprising: 

-means for filtering the blocks of the decoded coeffi- 
cients in two spatial directions. 
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